Papers with compression ratio
EASSE: Easier Automatic Sentence Simplification Evaluation (D19-3)
Copied to clipboard
| Challenge: | EASSE provides access to a broad range of evaluation resources including standard automatic metrics, word-level accuracy scores and reference-independent quality estimation features. |
| Approach: | They propose to provide a Python package that provides access to automatic evaluation and comparison of Sentence Simplification (SS) systems. |
| Outcome: | The proposed tool allows comparison and understanding of the performance of Sentence Simplification (SS) systems. |
TokLens: A Multilingual Lens on Tokenizer Quality for LLMs (2026.acl-srw)
Copied to clipboard
| Challenge: | TokLens is an open-source toolkit for evaluating tokenizer quality across languages . authors evaluated 24 tokenizers from major LLM families across 15 typologically diverse languages - a gap that is stark in Japanese . |
| Approach: | They evaluate 24 tokenizers from major LLM families across 15 typologically diverse languages and correlate these metrics with downstream performance. |
| Outcome: | The proposed tokenizers produce 56x more tokens per word in Japanese than in English . the newer tokenizer Qwen2.5 and Gemma-2 reduce this gap to under 4x . |
DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization (2022.acl-short)
Copied to clipboard
Zheng Li, Zijian Wang, Ming Tan, Ramesh Nallapati, Parminder Bhatia, Andrew Arnold, Bing Xiang, Dan Roth
| Challenge: | Empirical analyses show that pre-trained sequence-to-sequence models can achieve a 16.5x model footprint compression ratio with little performance drop relative to full-precision counterparts. |
| Approach: | They propose to distill and quantize pre-trained sequence-to-sequence models to reduce memory and latency requirements. |
| Outcome: | Empirical results show that the proposed model achieves 16.5x model footprint compression ratio with little performance drop relative to full-precision counterparts on multiple summarization and QA datasets. |
Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for converting visual tokens into tokens are limited by their high volume . et al., 2023; Zheng e.t., 2023): a revolution in video understanding. |
| Approach: | They propose a language-aware dynamic token compression system that converts video clips into soft caption tokens as visual representations. |
| Outcome: | The proposed method reduces FLOPs by 49% while maintaining competitive performance. |
Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Spectral-normalized identity priors (SNIP) is a structured pruning approach for a Transformer model. |
| Approach: | They propose a structured pruning approach which penalizes an entire residual module toward an identity mapping. |
| Outcome: | The proposed method improves on 5 GLUE benchmark tasks while maintaining comparable performance. |
RelationalCoder: Rethinking Complex Tables via Programmatic Relational Transformation (2025.acl-long)
Copied to clipboard
| Challenge: | Semi-structured tables remain a major obstacle for automated data processing and analytics. |
| Approach: | They propose a technique called Loop Reference Decoding which identifies expandable groups and replicates each group using a concise loop over its repetitive region. |
| Outcome: | The proposed technique reduces output length from O(N M) to approximately O(K) Extensive experiments on HiTab and MultiHiertt show that it boosts Llama-2 and Mistral models by more than 20%, and GPT-4o by over 4%. |
With Measured Words: Simple Sentence Selection for Black-Box Optimization of Sentence Compression Algorithms (2021.eacl-main)
Copied to clipboard
| Challenge: | Sentence Compression is the task of generating a shorter, yet grammatical, version of a given sentence, preserving the essence of the original sentence. |
| Approach: | They propose a Black-Box Optimizer for Compression to find the best candidates for compression . they use a black-box compression algorithm to predict how well each sentence could be compressed . |
| Outcome: | The proposed algorithm improves both accuracy and Rouge-F1-score on three datasets. |
Reference-free Summarization Evaluation via Semantic Correlation and Compression Ratio (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for summarization use human annotations as reference. |
| Approach: | They propose a new automatic reference-free evaluation metric that compares semantic distribution between source document and summary by pretrained language models and considers summary compression ratio. |
| Outcome: | The proposed metric is more consistent with human evaluation in terms of coherence, consistency, relevance and fluency. |
SP3: Enhancing Structured Pruning via PCA Projection (2024.findings-acl)
Copied to clipboard
| Challenge: | Structured pruning is a widely used technique for reducing the size of pre-trained language models, but current methods overlook the potential of compressing the hidden dimension d in PLMs. |
| Approach: | They propose a structured pruning approach that projectes features into a space defined by principal components before masking the hidden dimension d in pre-trained language models. |
| Outcome: | Experiments on benchmarks show that SP3 can reduce d by 70%, compress 94% of the BERTbase model, and maintain over 96% accuracy. |
Binarized LSTM Language Model (N18-1)
Copied to clipboard
| Challenge: | Long short-term memory (LSTM) language models are widely used for automatic speech recognition and natural language processing (NLP) however, they are limited by the word embedding layer. |
| Approach: | They propose to encode words into binary vectors and use binarized LSTM parameters to achieve high memory compression. |
| Outcome: | The proposed model achieves 11.3 compression ratio without loss of performance and 31.6 compression ratio with acceptable performance degradation. |
BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization (P19-1)
Copied to clipboard
| Challenge: | Existing text summarization datasets are compiled from news articles, where summary-worthy content often appears in the beginning of input articles. |
| Approach: | They present a novel dataset, BIGPATENT, consisting of 1.3 million records of U.S. patent documents along with human written abstractive summaries. |
| Outcome: | The proposed dataset is compared with existing summarization datasets and demonstrates that salient content is evenly distributed in the input. |
AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation (2024.findings-emnlp)
Copied to clipboard
Jia Fu, Xiaoting Qin, Fangkai Yang, Lu Wang, Jue Zhang, Qingwei Lin, Yubo Chen, Dongmei Zhang, Saravan Rajmohan, Qi Zhang
| Challenge: | Recent advances in Large Language Models have transformed ML/AI development . a reevaluation of AutoML principles for Retrieval-Augmented Generation (RAG) systems is needed. |
| Approach: | They propose a framework for hyper-parameter tuning and a hierarchical MAB method for efficient exploration of large search spaces. |
| Outcome: | The proposed framework outperforms baseline methods in more challenging optimization scenarios. |
Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization (2024.findings-emnlp)
Copied to clipboard
Yixin Ji, Yang Xiang, Juntao Li, Qingrong Xia, Zi Ye, Xinyu Duan, Zhefeng Wang, Kehai Chen, Min Zhang
| Challenge: | Large language models require a balance between efficiency and performance. |
| Approach: | They propose a low-rank compression technique that reduces non-essential parameters by decomposing weight matrices into products of two low-ranked matrici. |
| Outcome: | The proposed method outperforms existing pruning and low-rank compression techniques in maintaining model performance at the same compression ratio. |
AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Models (2022.findings-emnlp)
Copied to clipboard
Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeongwook Kim, Jung-Woo Ha, Nako Sung, Dongsoo Lee
| Challenge: | Existing approaches to improve inference efficiency by accelerating model fine-tuning have not been thoroughly explored. |
| Approach: | They propose to combine parameter-efficient adaptation and model compression to accelerate model . they propose to freeze binary parameters and scale scaling factors for target tasks . |
| Outcome: | The proposed algorithm achieves >10x compression ratio under 4-bit quantization and >1,000x reduction in trainable parameters. |
Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning (2026.eacl-long)
Copied to clipboard
| Challenge: | Curriculum learning has improved efficiency across machine learning domains, but remains underexplored for language model pretraining. |
| Approach: | They present a systematic investigation of curriculum learning in LLM pretraining . they use vanilla curriculum learning, pacing-based sampling, and interleaved curricula . |
| Outcome: | The proposed framework accelerates convergence in early and mid-training phases, reducing training steps by 18-45% to reach baseline performance. |
Efficient Learned Data Compression via Dual-Stream Feature Decoupling (2026.acl-long)
Copied to clipboard
| Challenge: | Learned data compression has achieved superior compression ratios, but balancing precise probability modeling with system efficiency remains challenging. |
| Approach: | They propose a Dual-Stream Multi-Scale Decoupler that disentangles local and global contexts to replace deep serial processing with shallow parallel streams. |
| Outcome: | The proposed method achieves state-of-the-art performance in both compression ratio and throughput while maintaining the lowest latency and memory usage. |
SCA: Selective Compression Attention for Efficiently Extending the Context Window of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to compress the KV cache of large language models are expensive and limited in their context window and cost. |
| Approach: | They propose a method to expand the context window and reduce memory footprint by compressing the KV cache of large language models. |
| Outcome: | The proposed method can reduce memory footprint and expand context window of large language models without training. |
Cross-Lingual Sentence Compression for Length-Constrained Subtitles in Low-Resource Settings (2025.coling-main)
Copied to clipboard
| Challenge: | a new system for sentence compression is developed for broadcast and live media . the goal is to reduce the on-screen visual constraints of the text . |
| Approach: | They develop a machine translation and sentence compression system that trains on openly available parallel corpora organized by compression ratios. |
| Outcome: | The proposed system preserves high semantic meaning and metric evaluations for compressed contexts. |
Inverse Reinforcement Learning for Text Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that inverse reinforcement learning (RL) training has certain disadvantages such as object mismatch and exposure bias. |
| Approach: | They propose inverse reinforcement learning (IRL) as an effective paradigm for training abstractive summarization models. |
| Outcome: | The proposed model outperforms MLE and RL baselines on ROUGE, coverage, novelty, compression ratio, factuality, and human evaluations. |
Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated exceptional performance across diverse natural language processing tasks. |
| Approach: | They propose a method for converting multi-head attention into grouped-query attention with any compression ratio of KV heads. |
| Outcome: | The proposed method can compress up to 87.5% KV heads of LLaMA2-7B model and 75% Kv heads of Sheared-LLa MA-1.3B with acceptable performance degradation. |
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization (2021.acl-long)
Copied to clipboard
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, Weizhu Chen
| Challenge: | 'lottery tickets' can be trained to match the performance of a full model . subnetwork training can also outperform random sampled subnetworks of the same size . |
| Approach: | They propose to train a subnetwork of 'lottery tickets' to match the full model's performance. |
| Outcome: | The proposed model outperforms subnetworks of the same size in a phase transition phenomenon . the proposed model improves single task fine-tuning by 0.9 points on BERT-base and 1.0 points on GLUE large . |
Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations (2024.findings-acl)
Copied to clipboard
| Challenge: | Structured pruning is a feasible solution for end-side LLM deployment . however, achieving a high compression ratio for scaled-up LLMs remains a challenge . |
| Approach: | They propose a task-agnostic structured pruning approach coupled with a compact Transformer architecture to prune LLMs into an intra-module low-rank architecture. |
| Outcome: | The proposed approach reduces transitional activations inside multi-head attention (MHA) and multi-layer perceptron (MLP) modules while preserving inter-module activations sensitive to perturbations. |
Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation (2022.emnlp-main)
Copied to clipboard
| Challenge: | a new framework for sentence summarization is available that can be trained reference-free . a high-quality dataset of sentence-summary pairs with varying degrees of compression ratios is obtained . |
| Approach: | They propose a framework for sentence summarization that can be trained reference-free . they propose 'referee' that iteratively distills latent knowledge into better models . |
| Outcome: | The proposed framework outperforms existing models in the use of explicit examples from teacher models without compromising the quality of the summarization. |
Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained sequence-to-sequence models have advanced the state of the art on text generation tasks. |
| Approach: | They introduce a modular encoder-decoder framework for flexible sequence-to-sequence model compression. |
| Outcome: | The proposed framework can achieve flexible compression ratios from 1.1x to 6x with little to moderate relative performance drop. |
Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Styles (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt compression reduces inference time and costs while maintaining informativeness for different usage scenarios. |
| Approach: | They propose a framework that adapts a smaller language model to compress prompts for a larger model on a new task without additional training. |
| Outcome: | The proposed framework outperforms two baseline models in four tasks . iteratively generates and selects effective compressed prompts as task-specific demonstrations . |
ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Recent approaches to reduce resource requirements for task-specific large language models have been developed. |
| Approach: | They propose a delta compression approach that optimizes for importance of a model . they use SVD to dynamically adjust the sparsity ratios of different vectors based on their importance . |
| Outcome: | The proposed approach achieves state-of-the-art in retaining task-specific knowledge even at high sparsity ratios. |
Understanding and Improving Information Preservation in Prompt Compression for LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large language models have enabled their successful application to a broad range of tasks. |
| Approach: | They propose a framework that allows for in-depth analysis of prompt compression methods. |
| Outcome: | The proposed framework analyzes state-of-the-art soft and hard compression methods . it shows that some fail to preserve key details from the original prompt, limiting performance on complex tasks. |
MiniKV: Pushing the Limits of 2-Bit KV Cache via Compression and System Co-Design for Efficient Long Context Inference (2025.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art 2-bit KV cache quantization methods achieve excellent results in accelerating LLM inference while retaining accuracy on long context tasks. |
| Approach: | They propose a method based on 2-bit KV cache quantization with adaptive KV policies that retain LLM accuracy with only a subset of KV states. |
| Outcome: | The proposed method outperforms state-of-the-art methods on a wide range of long context tasks while retaining accuracy. |
Concise and Precise Context Compression for Tool-Using Language Models (2024.findings-acl)
Copied to clipboard
Yang Xu, Yunlong Feng, Honglin Mu, Yutai Hou, Yitong Li, Xinghao Wang, Wanjun Zhong, Zhongyang Li, Dandan Tu, Qingfu Zhu, Min Zhang, Wanxiang Che
| Challenge: | Existing methods suffer from key information loss and difficulty in adjusting the length of compressed sequences based on documentation lengths. |
| Approach: | They propose two strategies for compressing tool documentation into concise and precise summary sequences for tool-using language models. |
| Outcome: | The proposed approach achieves comparable performance to the upper-bound baseline under 16x compression ratio. |
Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging (2024.emnlp-main)
Copied to clipboard
Deyuan Liu, Zhanyue Qin, Hairu Wang, Zhao Yang, Zecheng Wang, Fangying Rong, Qingbin Liu, Yanchao Hao, Bo Li, Xi Chen, Cunhang Fan, Zhao Lv, Dianhui Chu, Zhiying Tu, Dianbo Sui
| Challenge: | Existing methods for parameter pruning fail to utilize the knowledge from pruned parameters. |
| Approach: | They propose a method that uses manifold learning and the Information Bottleneck measure to merge similar layers to preserve model performance. |
| Outcome: | The proposed method outperforms pruning methods on multiple datasets and LLMs with quantization and achieves substantial compression ratios. |
GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have demonstrated that many layers are functionally redundant in large language models (LLMs), enabling model compression by removing these layers to reduce inference cost. |
| Approach: | They propose a framework that removes redundant layers to reduce inference cost by preserving sensitivity-aware singular values. |
| Outcome: | The proposed framework outperforms existing methods in 90% of the original model under a 20% compression ratio. |
Break Through the Compression Bottleneck: From Theory to Practice (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing compression methods suffer from bottleneck issues when compression ratio is increased. |
| Approach: | They propose a novel approach to combine low-rank decomposition and quantization methods to reduce the compression bottleneck. |
| Outcome: | The proposed method reduces the computational and memory overhead of existing methods while maintaining model accuracy. |